Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move to dag_run.logical_date from execution date in OpenLineage provider #41889

Merged

Conversation

mobuchowski
Copy link
Contributor

Remove usage of deprecated execution_date in favor of logical_date

@potiuk
Copy link
Member

potiuk commented Sep 1, 2024

Is it backwards-compatible with Airflow 2?

@mobuchowski
Copy link
Contributor Author

@potiuk logical_date property was added back in 2021

Screenshot 2024-09-02 at 11 14 00

@potiuk
Copy link
Member

potiuk commented Sep 2, 2024

logical_date teproperty was added back in 2021

I know, just checking if it is ok to change it in the provider - which will be (for the next 6 months or so) only installed on Airflow 2 :)

@potiuk
Copy link
Member

potiuk commented Sep 2, 2024

Simply execution_date and logical_date have different semantics, so my question is, if that will impact observed open-lineage events. And if so - whether it should be somehow explained in the changelog so that users are not surprised.

@mobuchowski
Copy link
Contributor Author

@potiuk does it really have different semantics?

From what I can find, it's essentially a rename:

From AIP-39

We need a single date value that is the "canonical" date for the DagRun date, that can be used for sorting DagRuns for the UI view, so the current "execution_date" column on DagRun needs to be kept, but we should rename - and we've chosen "logical_date".

Also in the docstring:

This replaces execution_date in Airflow 2.1 and prior. The idea is essentially the same, just a different name.

Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason.

@potiuk
Copy link
Member

potiuk commented Sep 2, 2024

Basically, we want the same: the time when the DagRun is scheduled to be run, without consideration if it's actually going to run at that time, due to scheduler being backed up or other reason.

Yep. You are right. I had confused it with "start/end interval" and other macros available in context, but logical_date and execution_date are really aliases.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski force-pushed the move-to-logical-date-ol-provider branch from 76e9a97 to d38e89c Compare September 2, 2024 11:55
@mobuchowski mobuchowski merged commit 8640f3e into apache:main Sep 2, 2024
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants